{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "text_attack_keras_example",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GfiQHWweOb-C"
      },
      "source": [
        "# TextAttack on Keras Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/QData/TextAttack/blob/master/docs/2notebook/Example_6_Keras.ipynb)\n",
        "\n",
      "[![View Source on GitHub](https://img.shields.io/badge/github-view%20source-black.svg)](https://github.com/QData/TextAttack/blob/master/docs/2notebook/Example_6_Keras.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wv850rJPPE99"
      },
      "source": [
        "## Training\n",
        "\n",
        "The code below trains a basic neural network on a series of movie reviews from the IMDB dataset, loaded using Tensorflow's datasets module. Each review is encoded as a sequence of tokens corresponding to a word's index in the vocabulary. Class labels are provided, denoting a positive or negative sentiment. \n",
        "\n",
        "See [here](https://www.tensorflow.org/api_docs/python/tf/keras/datasets/imdb/load_data) for more information on the IMDB dataset. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Oru-kBljwyqd",
        "outputId": "4c2a2585-edd0-4a7e-a7cf-53acfb64076d"
      },
      "source": [
        "import tensorflow as tf\n",
        "import keras\n",
        "import os\n",
        "import numpy as np\n",
        "from keras.utils import to_categorical\n",
        "import torch\n",
        "import textattack\n",
        "from textattack.models.wrappers import ModelWrapper\n",
        "from textattack.datasets import HuggingFaceDataset\n",
        "from textattack.attack_recipes import PWWSRen2019\n",
        "\n",
        "\n",
        "import matplotlib\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "import numpy as np\n",
        "from keras.utils import to_categorical\n",
        "from keras import models\n",
        "from keras import layers\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "from keras.layers import Flatten\n",
        "from keras.layers import Dropout"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[34;1mtextattack\u001b[0m: Downloading https://textattack.s3.amazonaws.com/word_embeddings/paragramcf.\n",
            "100%|██████████| 481M/481M [00:16<00:00, 29.8MB/s]\n",
            "\u001b[34;1mtextattack\u001b[0m: Unzipping file /root/.cache/textattack/tmpdmzgnt8b.zip to /root/.cache/textattack/word_embeddings/paragramcf.\n",
            "\u001b[34;1mtextattack\u001b[0m: Successfully saved word_embeddings/paragramcf to cache.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QUT675TAVVle"
      },
      "source": [
        "Below, we load the IMDB dataset from Tensorflow and transform it for our classifier, using a Bag-of-Words format. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6rR709EZvO6O",
        "outputId": "7c57f56a-d0bd-46b8-eec4-bd1be1966583"
      },
      "source": [
        "\n",
        "NUM_WORDS = 10000\n",
        "\n",
        "(x_train_tokens, y_train), (x_test_tokens, y_test) = tf.keras.datasets.imdb.load_data(\n",
        "    path=\"imdb.npz\",\n",
        "    num_words=NUM_WORDS,\n",
        "    skip_top=0,\n",
        "    maxlen=None,\n",
        "    seed=113,\n",
        "    start_char=1,\n",
        "    oov_char=2,\n",
        "    index_from=3\n",
        ")\n",
        "\n",
        "\n",
        "def transform(x):\n",
        "  x_transform = []\n",
        "  for i, word_indices in enumerate(x):\n",
        "    BoW_array = np.zeros((NUM_WORDS,))\n",
        "    for index in word_indices:\n",
        "      if index < len(BoW_array):\n",
        "        BoW_array[index] += 1\n",
        "    x_transform.append(BoW_array)\n",
        "  return np.array(x_transform)\n",
        "    \n",
        "\n",
        "index = int(0.9 * len(x_train_tokens))\n",
        "x_train = transform(x_train_tokens)[:index]\n",
        "x_test = transform(x_test_tokens)[index:]\n",
        "y_train = np.array(y_train[:index])\n",
        "y_test = np.array(y_test[index:])\n",
        "y_train = to_categorical(y_train)\n",
        "y_test = to_categorical(y_test)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb.npz\n",
            "17465344/17464789 [==============================] - 0s 0us/step\n",
            "(22500, 10000) (22500, 2) (2500, 10000) (2500, 2)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7DpUFR0fVmzz"
      },
      "source": [
        "With our data successfully loaded, we can now design and trained our model. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YjLVCp3Z0SUj",
        "outputId": "786c1a9b-8ee6-4ede-942a-17263ffc2aa0"
      },
      "source": [
        "\n",
        "#Model Created with Keras\n",
        "model = Sequential()\n",
        "model.add(Dense(512, activation='relu', input_dim=NUM_WORDS))\n",
        "model.add(Dropout(0.3))\n",
        "model.add(Dense(100, activation='relu'))\n",
        "model.add(Dense(2, activation='sigmoid'))\n",
        "\n",
        "\n",
        "\n",
        "opt = keras.optimizers.Adam(learning_rate=0.00001)\n",
        "\n",
        "model.compile(\n",
        " optimizer = opt,\n",
        " loss = \"binary_crossentropy\",\n",
        " metrics = [\"accuracy\"]\n",
        ")\n",
        "\n",
        "\n",
        "results = model.fit(\n",
        " x_train, y_train,\n",
        " epochs= 18,\n",
        " batch_size = 512,\n",
        " validation_data = (x_test, y_test)\n",
        ")\n",
        "\n",
        "\n",
        "print(results.history)\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<bound method Model.summary of <tensorflow.python.keras.engine.sequential.Sequential object at 0x7fa0458fee10>>\n",
            "Epoch 1/18\n",
            "44/44 [==============================] - 3s 28ms/step - loss: 0.7069 - accuracy: 0.5209 - val_loss: 0.6731 - val_accuracy: 0.6240\n",
            "Epoch 2/18\n",
            "44/44 [==============================] - 1s 16ms/step - loss: 0.6760 - accuracy: 0.6041 - val_loss: 0.6543 - val_accuracy: 0.6840\n",
            "Epoch 3/18\n",
            "44/44 [==============================] - 1s 16ms/step - loss: 0.6597 - accuracy: 0.6453 - val_loss: 0.6353 - val_accuracy: 0.7192\n",
            "Epoch 4/18\n",
            "44/44 [==============================] - 1s 16ms/step - loss: 0.6378 - accuracy: 0.6942 - val_loss: 0.6152 - val_accuracy: 0.7464\n",
            "Epoch 5/18\n",
            "44/44 [==============================] - 1s 16ms/step - loss: 0.6181 - accuracy: 0.7217 - val_loss: 0.5942 - val_accuracy: 0.7808\n",
            "Epoch 6/18\n",
            "44/44 [==============================] - 1s 24ms/step - loss: 0.5933 - accuracy: 0.7553 - val_loss: 0.5729 - val_accuracy: 0.8008\n",
            "Epoch 7/18\n",
            "44/44 [==============================] - 1s 16ms/step - loss: 0.5719 - accuracy: 0.7759 - val_loss: 0.5507 - val_accuracy: 0.8148\n",
            "Epoch 8/18\n",
            "44/44 [==============================] - 1s 16ms/step - loss: 0.5450 - accuracy: 0.7960 - val_loss: 0.5290 - val_accuracy: 0.8256\n",
            "Epoch 9/18\n",
            "44/44 [==============================] - 1s 16ms/step - loss: 0.5201 - accuracy: 0.8148 - val_loss: 0.5064 - val_accuracy: 0.8336\n",
            "Epoch 10/18\n",
            "44/44 [==============================] - 1s 16ms/step - loss: 0.5031 - accuracy: 0.8212 - val_loss: 0.4850 - val_accuracy: 0.8388\n",
            "Epoch 11/18\n",
            "44/44 [==============================] - 1s 16ms/step - loss: 0.4799 - accuracy: 0.8326 - val_loss: 0.4646 - val_accuracy: 0.8468\n",
            "Epoch 12/18\n",
            "44/44 [==============================] - 1s 16ms/step - loss: 0.4556 - accuracy: 0.8456 - val_loss: 0.4455 - val_accuracy: 0.8504\n",
            "Epoch 13/18\n",
            "44/44 [==============================] - 1s 16ms/step - loss: 0.4328 - accuracy: 0.8578 - val_loss: 0.4276 - val_accuracy: 0.8548\n",
            "Epoch 14/18\n",
            "44/44 [==============================] - 1s 16ms/step - loss: 0.4168 - accuracy: 0.8623 - val_loss: 0.4116 - val_accuracy: 0.8568\n",
            "Epoch 15/18\n",
            "44/44 [==============================] - 1s 16ms/step - loss: 0.3980 - accuracy: 0.8702 - val_loss: 0.3976 - val_accuracy: 0.8624\n",
            "Epoch 16/18\n",
            "44/44 [==============================] - 1s 16ms/step - loss: 0.3845 - accuracy: 0.8696 - val_loss: 0.3850 - val_accuracy: 0.8620\n",
            "Epoch 17/18\n",
            "44/44 [==============================] - 1s 16ms/step - loss: 0.3730 - accuracy: 0.8735 - val_loss: 0.3752 - val_accuracy: 0.8660\n",
            "Epoch 18/18\n",
            "44/44 [==============================] - 1s 16ms/step - loss: 0.3523 - accuracy: 0.8813 - val_loss: 0.3646 - val_accuracy: 0.8668\n",
            "{'loss': [0.6982908248901367, 0.6719189882278442, 0.6552321314811707, 0.6339828372001648, 0.6116981506347656, 0.5892458558082581, 0.5651863813400269, 0.5425485968589783, 0.5191691517829895, 0.49679040908813477, 0.47490108013153076, 0.4527125358581543, 0.4317401349544525, 0.4114888310432434, 0.39577117562294006, 0.381359338760376, 0.3658391833305359, 0.3557420074939728], 'accuracy': [0.5377333164215088, 0.6135555505752563, 0.6563555598258972, 0.6990666389465332, 0.731333315372467, 0.7592889070510864, 0.7805333137512207, 0.7969777584075928, 0.8121333122253418, 0.8241778016090393, 0.8349778056144714, 0.8455111384391785, 0.8550222516059875, 0.8615111112594604, 0.8684889078140259, 0.8726222515106201, 0.8766666650772095, 0.8783110976219177], 'val_loss': [0.673081636428833, 0.6542609930038452, 0.6353172063827515, 0.6151610612869263, 0.5941841006278992, 0.5729244947433472, 0.5507248044013977, 0.5290232300758362, 0.5063891410827637, 0.4850437641143799, 0.46456265449523926, 0.44545814394950867, 0.42756685614585876, 0.41164901852607727, 0.39760521054267883, 0.38501957058906555, 0.37516292929649353, 0.3646129071712494], 'val_accuracy': [0.6240000128746033, 0.6840000152587891, 0.7192000150680542, 0.746399998664856, 0.7807999849319458, 0.8008000254631042, 0.8148000240325928, 0.8256000280380249, 0.8335999846458435, 0.8388000130653381, 0.8468000292778015, 0.8503999710083008, 0.8547999858856201, 0.8568000197410583, 0.8623999953269958, 0.8619999885559082, 0.8659999966621399, 0.8668000102043152]}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N5OiEIQFSqCR"
      },
      "source": [
        "## Attacking\n",
        "\n",
        "With our model trained, we can create a  `ModelWrapper` that will allow us to run TextAttack on a custom Keras model. Each `ModelWrapper` must implement a single method, `__call__`, which takes a list of strings and returns a `List`, `np.ndarray`, or `torch.Tensor` of predictions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W61gKHFs6Wj0",
        "outputId": "bc370c37-671c-470f-95db-a4e378610099"
      },
      "source": [
        "\n",
        "\n",
        "class CustomKerasModelWrapper(ModelWrapper):\n",
        "    def __init__(self, model):\n",
        "        self.model = model\n",
        "\n",
        "    def __call__(self, text_input_list):\n",
        "      text_array = np.array([words2tokens(text_input) for text_input in text_input_list])\n",
        "      prediction = self.model.predict(text_array)\n",
        "      preds = [list(prediction[i][0]) for i in range(0, len(prediction))]\n",
        "      return preds\n",
        "\n",
        "\n",
        "CustomKerasModelWrapper(model)([\"the movie was awful\", \"the movie was awesome\"])\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[[0.55557764, 0.44437635], [0.4965529, 0.50741225]]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wz7MCIx9THHT"
      },
      "source": [
        "With our `ModelWrapper` constructed, we can use TextAttack's HuggingFaceDataset module to load reviews for testing, alongside TextAttack's PWWSRen2019 module to serve as our attack recipe. \n",
        "\n",
        "The attack below leverages TextAttack's `Attack` class, capable of running attacks against entire datasets. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1NQusSfN40aK",
        "outputId": "dc2975b1-ff96-4bf3-f94f-e59ed22ec858"
      },
      "source": [
        "model_wrapper = CustomKerasModelWrapper(model)\n",
        "dataset = HuggingFaceDataset(\"rotten_tomatoes\", None, \"test\", shuffle=True)\n",
        "\n",
        "attack = PWWSRen2019.build(model_wrapper)\n",
        "\n",
        "results_iterable = attack.attack_dataset(dataset, indices=range(10))\n",
        "for result in results_iterable:\n",
        "  print()\n",
        "  print()\n",
        "  print(result.__str__(color_method='ansi'))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using custom data configuration default\n",
            "Reusing dataset rotten_tomatoes_movie_review (/root/.cache/huggingface/datasets/rotten_tomatoes_movie_review/default/1.0.0/9198dbc50858df8bdb0d5f18ccaf33125800af96ad8434bc8b829918c987ee8a)\n",
            "\u001b[34;1mtextattack\u001b[0m: Loading \u001b[94mdatasets\u001b[0m dataset \u001b[94mrotten_tomatoes\u001b[0m, split \u001b[94mtest\u001b[0m.\n",
            "\u001b[34;1mtextattack\u001b[0m: Unknown if model of class <class 'tensorflow.python.keras.engine.sequential.Sequential'> compatible with goal function <class 'textattack.goal_functions.classification.untargeted_classification.UntargetedClassification'>.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\u001b[92mPositive (50%)\u001b[0m --> \u001b[37m[SKIPPED]\u001b[0m\n",
            "\n",
            "movies like high crimes flog the dead horse of surprise as if it were an obligation . how about surprising us by trying something new ?\n",
            "\n",
            "\n",
            "\u001b[92mPositive (53%)\u001b[0m --> \u001b[37m[SKIPPED]\u001b[0m\n",
            "\n",
            "in a 102-minute film , aaliyah gets at most 20 minutes of screen time . . . . most viewers will wish there had been more of the \" queen \" and less of the \" damned . \"\n",
            "\n",
            "\n",
            "\u001b[91mNegative (51%)\u001b[0m --> \u001b[92mPositive (50%)\u001b[0m\n",
            "\n",
            "more \u001b[91mlikely\u001b[0m to have you \u001b[91mscratching\u001b[0m your \u001b[91mhead\u001b[0m than hiding under your seat .\n",
            "\n",
            "more \u001b[92mprobably\u001b[0m to have you \u001b[92mcancel\u001b[0m your \u001b[92mpsyche\u001b[0m than hiding under your seat .\n",
            "\n",
            "\n",
            "\u001b[91mNegative (50%)\u001b[0m --> \u001b[92mPositive (51%)\u001b[0m\n",
            "\n",
            "\u001b[91mslow\u001b[0m , silly and unintentionally hilarious .\n",
            "\n",
            "\u001b[92measy\u001b[0m , silly and unintentionally hilarious .\n",
            "\n",
            "\n",
            "\u001b[92mPositive (50%)\u001b[0m --> \u001b[91mNegative (52%)\u001b[0m\n",
            "\n",
            "by its modest , straight-ahead standards , undisputed scores a direct \u001b[92mhit\u001b[0m .\n",
            "\n",
            "by its modest , straight-ahead standards , undisputed scores a direct \u001b[91mshoot\u001b[0m .\n",
            "\n",
            "\n",
            "\u001b[92mPositive (51%)\u001b[0m --> \u001b[37m[SKIPPED]\u001b[0m\n",
            "\n",
            "novak contemplates a heartland so overwhelmed by its lack of purpose that it seeks excitement in manufactured high drama .\n",
            "\n",
            "\n",
            "\u001b[91mNegative (57%)\u001b[0m --> \u001b[92mPositive (50%)\u001b[0m\n",
            "\n",
            "i walked away not really \u001b[91mknow\u001b[0m who \" they \" were , what \" they \" \u001b[91mlooked\u001b[0m \u001b[91mlike\u001b[0m . why \" they \" were here and what \" they \" \u001b[91mwanted\u001b[0m and quite \u001b[91mhonestly\u001b[0m , i didn't \u001b[91mcare\u001b[0m .\n",
            "\n",
            "i walked away not really \u001b[92mlove\u001b[0m who \" they \" were , what \" they \" \u001b[92msearch\u001b[0m \u001b[92mcomparable\u001b[0m . why \" they \" were here and what \" they \" \u001b[92mdesire\u001b[0m and quite \u001b[92maboveboard\u001b[0m , i didn't \u001b[92mattention\u001b[0m .\n",
            "\n",
            "\n",
            "\u001b[91mNegative (53%)\u001b[0m --> \u001b[92mPositive (51%)\u001b[0m\n",
            "\n",
            "for this sort of \u001b[91mthing\u001b[0m to work , we need agile performers , but the proficient , \u001b[91mdull\u001b[0m sorvino has no light touch , and rodan is out of his league .\n",
            "\n",
            "for this sort of \u001b[92mmatter\u001b[0m to work , we need agile performers , but the proficient , \u001b[92mblunt\u001b[0m sorvino has no light touch , and rodan is out of his league .\n",
            "\n",
            "\n",
            "\u001b[91mNegative (52%)\u001b[0m --> \u001b[92mPositive (50%)\u001b[0m\n",
            "\n",
            ". . . hopefully it'll be at the \u001b[91mdollar\u001b[0m theatres by the time christmas \u001b[91mrolls\u001b[0m \u001b[91maround\u001b[0m . \u001b[91mwait\u001b[0m to see it then .\n",
            "\n",
            ". . . hopefully it'll be at the \u001b[92mclam\u001b[0m theatres by the time christmas \u001b[92mroll\u001b[0m \u001b[92mround\u001b[0m . \u001b[92mwaiting\u001b[0m to see it then .\n",
            "\n",
            "\n",
            "\u001b[92mPositive (52%)\u001b[0m --> \u001b[91mNegative (50%)\u001b[0m\n",
            "\n",
            "like \u001b[92mkissing\u001b[0m jessica stein , amy's orgasm has a \u001b[92mkey\u001b[0m strength in its willingness to \u001b[92mexplore\u001b[0m its principal characters with honesty , \u001b[92minsight\u001b[0m and humor .\n",
            "\n",
            "like \u001b[91mkiss\u001b[0m jessica stein , amy's orgasm has a \u001b[91mpaint\u001b[0m strength in its willingness to \u001b[91mresearch\u001b[0m its principal characters with honesty , \u001b[91mperceptiveness\u001b[0m and humor .\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "se8V4zjqUiji"
      },
      "source": [
        "## Conclusion\n",
        "\n",
        "Great! We trained a binary classifier, created a custom `ModelWrapper` for Keras models, and successsfully ran adversarial attacks against our trained Keras model! This serves a basic demo for how to use TextAttack within your own environments. \n"
      ]
    }
  ]
}